The present invention relates to electronic data storage in which data is copied or xe2x80x9cmirroredxe2x80x9d from a local mass data storage system to a remote mass data storage system for the purposes of backup, security and recovery in the event of a catastrophic failure. More particularly, the present invention relates to a new and improved method and data structure by which to verify data and check parity consistency between the local and remote mass data storage systems by communicating only a relatively small fraction of the information previously required between the local and remote systems, thereby increasing the capacity or bandwidth for performing ongoing, non-verification storage operations.
Modern data processing systems recognize the imperative nature of preserving and maintaining the correct data for computational use. In some types of data processing systems, such as those used in banks, financial institutions and other organizations which deal with sensitive and critical data, it is absolutely necessary to guard against massive inadvertent data errors and data loss due to equipment malfunctions, system failures and natural disasters. One common way to avoid such catastrophic errors is to copy or xe2x80x9cmirrorxe2x80x9d the complete data stored at an active data processing system to a mirroring data processing system. The mirroring data processing system is located at a geographically remote location. If the active system experiences a catastrophic failure, the mirroring system has all of the data and computational capabilities necessary to assume responsibility for performing the primary data processing functions as a substitute for the failed active system. Conversely, if the monitoring system fails, the active system proceeds in its normal fashion to continue executing the primary computational and data storage functions. As soon as possible after the catastrophe, the failed one of the remote or active systems is again made functional to restore the backup, security and recovery capabilities of the system.
Of course, the functional integrity of such a mirrored mass data storage system depends entirely on the ability to maintain an accurate copy of the data from the active system on the mirroring system, assuming that both systems have comparable computational capabilities. As the active system adds new data to its active mass data storage system, or modifies its existing data, a copy of that new or changed data must be transferred to and stored on the mirroring data processing system on a frequent basis.
Updating the data of the mirroring mass data storage system may occur synchronously or asynchronously. Synchronous updates are performed approximately simultaneously with the change or addition at the active mass data storage system. Synchronous updates occur continually during the use of the active data processing system. Asynchronous updates are performed at periodic time intervals, and involve collecting blocks of added or modified data into groups and communicating the groups at the time intervals. The asynchronous updates occur at relatively frequent intervals, since the integrity of the mirroring system depends maintaining the state of the mirrored data the same as the state of the active data.
Complex mass data storage systems may involve storing tens or hundreds of terabytes of information. During the normal high-intensity use of such complex systems, a significant fraction of the total information will be updated on a continuing basis, either synchronously or asynchronously. The amount of information to be transferred as updates between the active and remote systems requires rapid communication capability between the active and remote systems. Consequently, the active system and geographically-remote mirroring systems are frequently linked by high-speed, high-capacity land lines and satellite links to communicate the data back and forth. Using lower-performance communication links is simply not effective from a performance standpoint, due to the high requirement for continuing updates involving substantial transfers of data between the active and mirroring systems. The cost of using high-speed, high-capacity land lines and satellite links becomes significant in implementing mirrored, redundant data backup systems.
In addition to communicating updates between the active and remote systems rapidly and with minimal time delay, another aspect of a reliable mirrored mass data storage system involves periodically performing disk verify operations or disk verify and parity check operations. A verify and check operation involves scanning the mass storage media, which is usually formed from multiple disk drives configured as a redundant array of independent disks (RAID), looking for disk defects and checking to determine that the stored data and its parity are consistent. Parity is a short mathematical surrogate or proxy for the data. The parity allows the data to be reconstructed, if necessary.
Occasionally a disk drive will degrade to the point that one of the divisions of its storage media, known as a sector, becomes defective. A defective sector is incapable of storing data reliably. The disk verify operation involves reading the data and writing it back, to determine proper functionality. The parity check is an optional and frequently-performed operation which is combined with the disk verify operation. The parity check involves computing the parity of the data read during the verify operation, and comparing the computed parity with the stored parity that was read with the associated data. Although unintended, it is occasionally possible that a disk drive or the associated mass data storage software will create a situation where the data and the stored parity do not match. A mismatch of the parity and data indicates a problem of data reliability, and such problems should be resolved quickly.
It is essential to detect and correct errors arising from a mismatch of the data and parity and from faulty disk drives and sectors while the remaining RAID mass storage system is properly functional. Although RAID mass data storage systems have been developed to permit certain errors to be detected and corrected, substantially all of the remaining, non-faulty components of the RAID system must usually be properly functional in order to correct the error. If multiple parts of the RAID system have failed or degraded, it may be extremely difficult or impossible to correct multiple errors. Thus, if the errors are not searched out and corrected on a reliable and prompt basis, the existence of such errors may prevent correction and data reconstruction. Verify and check operations are intended to seek out and correct such errors before additional errors occur.
The typical technique of performing the verify and check operation in a mirrored system is to periodically transfer entire blocks of data from the mirroring mass data storage system to the active mass data storage system where the data is compared. Transferring such massive amounts of data is expensive because of the cost of using the land lines or satellite links, and because transferring and comparing the data consumes substantial computational resources of the active data processing system. Diverting the computational resources of the active data processing system away from its normal processing activities reduces the overall performance of the active system and frequently requires the size and capacity of the active system to be increased to compensate for performing the transfer and compare activities.
The complexities and costs involved in transfer, compare, verify and check operations have led to the development of techniques to quantify the data for such operations, permitting such operations to be carried out in a convenient manner while simultaneously performing the normal data processing functions. One technique used is to divide the data into groups or blocks, and to perform such operations only on the blocks. The improvements from the two above identified previous inventions, of which the present invention is a continuation in part, involves a technique of effectively grouping data into blocks, combining the data with certain descriptive metadata in a data structure, and then performing activities with respect to each individual data structure. The data structure not only provides a convenient division of the data, but the metadata of the data structure can be effectively used to detect and recover from certain inadvertent errors, on a block-by-block basis or on a group-of-blocks basis. The data within these data structures may be reconstructed and verified in a minimal amount of time while consuming a minimal amount of computer resources. However, these previous inventions do not involve using these beneficial data structures and their data verification and reconstructive techniques for mirroring data between an active system and a geographically remote mirroring system.
The factors summarized above, and other background considerations not specifically mentioned but known within the field of mirrored mass data storage systems, have led to the present invention.
The present invention involves a method and a data structure which allows the data stored on an active mass data storage system and on the mirroring mass data storage system to be verified as consistent and reliable with a minimal requirement for communicating between the active system and the geographically-remote mirroring system. Another aspect of the invention involves reducing the cost of verifying the consistency of data in a mirrored mass data storage system, both by reducing the necessity for, and amount of communication over expensive land lines and satellite links, and by permitting more of the computational resources to be committed to active data processing operations rather than to verification and data transfer operations. An additional aspect of the invention involves performing verify and check operations between the active system and the mirroring mass data storage system without the need to transfer blocks of data between the systems, while still obtaining a relatively high degree of assurance that the data is consistent between the active and the mirroring systems. Yet another aspect of the invention is to utilize a data storage structure for storing and mirroring data on the active and mirroring systems which permits effective verification of the consistency of the data and the parity both on the local system (either the active or the mirroring system) and the remote other system.
The present invention makes advantageous use of a data structure which is an improved version of that data structure described in the two previous applications. The present invention utilizes the improved data structure to facilitate performing the verify and check operations locally, to thereby establish consistency between the locally stored data and parity. Part of the metadata generated by performing the local verify operation is a cyclical redundancy code (CRC). In accordance with the present invention, a time stamp is also included as part of the metadata of the improved data structure. The time stamp describes the time when the data of that data structure was originally written. The local CRC and time stamp information is communicated by the active system as a part of a mirroring verify and check command sent to the mirroring system. The mirroring system responds by performing a local verify and check operation to determine whether the data and parity of the corresponding mirroring data structures are locally consistent. Thereafter, the time stamp and the CRC which have been read locally are compared to the time stamp and the CRC communicated by the active system as part of the mirroring command. If the two time stamps and CRCs compare to be the same, there is a high degree of assurance that the data and parity on the active and mirroring systems are consistent with one another. If there is disagreement between the two time stamps and the two CRCs, the data with the most current (newest temporally) time stamp is used as the basis for updating the data having the least current (oldest temporally) time stamp.
The time stamp is used to determine which data (active or mirroring copy) to be used in correcting the error on the other system. The comparison of the CRC is used as an indication of whether the data has been erroneously modified or changed. If the CRCs are consistent, it is reliably indicated that the data has not been disturbed and that the data is consistent between the active and mirroring systems. This level of assurance is enhanced by the verify and check operations performed locally at both the active and the mirroring systems using the data structures, before the two time stamps and CRCs are compared.
An enhanced degree of assurance that the local data is accurate or can be corrected is thereby made possible without encountering the necessity, expense or diversion of computational resources involved in transferring substantial amounts of data between the active and mirroring systems to perform previous consistency verification operations. For example, an enhanced level of verification can be achieved in a mirrored system by transferring approximately 16 bytes of information compared to a prior requirement of transferring 4 Kbs of information for each block of data. In this example, the present invention permits a consistency verification operation to be performed by transferring approximately 0.4% of the information previously required to perform such a consistency verification operation on a mirrored mass data storage system.
A more complete appreciation of the present invention and its scope may be obtained from the accompanying drawings, which are briefly summarized below, from the following detail descriptions of presently preferred embodiments of the invention, and from the appended claims.